Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)

Reply to this post | Go Back
View Post [edit]

Poster: phdstudent Date: Nov 24, 2010 12:52pm
Forum: web Subject: Does a disallow in robots.txt actually delete archived content?

Suppose I have a web site without a robots.txt, which is archived by archive.org.

A year later, it shows up in the index and I add a robots.txt disallowing archive.org's crawler. Predictably, archive.org stops showing the archived page.

If I then remove the robots.txt, will archive.org resume showing that old archived version some time in the future, or has that page been permanently deleted, and will never be seen again?

Thank you.

Terms of Use (10 Mar 2001)